Proceedings  -  23rd  Annual  Conference  -  lEEE/EMBS  Oct.25-28,  2001,  Istanbul,  TURKEY 


Design  and  Implementation  of  Biologically  Realistic  Signal  to 

Symbol  Translators 

Jose  C.  Principe,  Vitor  Tavares 
University  of  Florida 

Computational  NeuroEngineering  Laboratory 
451  Engineering  Building,  PO  Box  116130,  Gainesville,  EL  32611-6130,  U.S.A. 


ABSTRACT 

This  paper  reviews  the  problem  of  translating  signals  into  sym¬ 
bols  preserving  maximally  the  information  contained  in  the  signal 
time  structure.  In  this  context,  we  motivate  the  use  of  nonconver- 
gent  dynamics  for  the  signal  to  symbol  translator.  We  then 
describe  a  biologically  realistic  model  of  the  olfactory  system  pro¬ 
posed  by  Walter  Freeman  that  has  locally  stable  dynamics  but  is 
globally  chaotic.  We  present  results  of  simulations  and  measure¬ 
ments  obtained  from  a  fabricated  analog  VLSI  chip. 

1.  INTRODUCTION 
1.  Introduction 


ated  by  a  dynamical  system  with  singularities  of  at  least  second 
order  (third  order  ODE).  A  chaotic  attractor  is  still  a  stable  repre¬ 
sentation,  might  exist  in  a  high  dimensional  space  (much  higher 
than  the  dimensionality  of  our  3D  world),  and  more  importantly 
its  dimensionality  can  be  controlled  by  parameters  of  the  SZTs. 
Forseeably  such  systems  are  capable  of  using  the  inner  structure 
of  trajectories  within  the  attractor  for  very  high  information  stor¬ 
age  and  rapid  recall,  but  we  still  do  not  fully  understand  how  to 
control  the  stability  of  recall  in  particular  in  the  presence  of  noise. 

Our  aim  is  to  construct  a  SET  that  operates  in  accordance 
with  the  neurodynamics  of  the  cerebral  cortex,  and  that  has  the 
sensitivity,  selectivity,  adaptiveness,  speed,  and  tolerance  of  noise 
that  characterizes  human  sensation.  We  will  be  using  a  model  of 
the  rabbit  olfactory  cortex  proposed  by  Walter  Freeman  to 
develop  and  implement  an  analog  VLSI  version  of  a  biologically 
realistic  SET  [1].  The  core  of  the  model  is  the  KII  network. 


There  are  many  important  differences  between  biological 
and  man-made  information  processing  systems.  Animals  have 
goal  driven  behavior  and  have  explored  inductive  principles 
throughout  the  course  of  evolution  to  work  reliably  in  a  nonGaus- 
sian,  nonstationary,  nonlinear  world.  Autonomous  man-made  sys¬ 
tems  with  sensors  and  computational  algorithms  (animats)  are  still 
unable  to  match  these  capabilities.  The  processing  device  that 
transforms  signals  into  symbols  is  called  here  the  signal-to-sym- 
bols  translator  (SET).  We  can  specify  an  optimal  SET  as  a  device 
that  is  able  to  capture  the  goal-relevant  information  contained  in 
the  signal  time  structure  and  map  it  with  as  little  excess  irrelevant 
information  as  possible  to  a  stable  representation  in  the  animat’s 
computational  framework. 

A  framework  where  the  SLTs  is  modeled  as  a  dynamical  sys¬ 
tem  coupled  to  the  external  world  seems  a  productive  alternative. 
We  will  center  the  discussion  in  distributed,  adaptive  arrange¬ 
ments  of  nonlinear  processing  elements  (PEs)  called  coupled  lat¬ 
tices  [3].  In  neurocomputing  content  addressable  memories 
(CAMs),  both  static  or  dynamic  (Hopfield  networks)  with  fixed 
points  have  been  proposed  and  shown  useful.  However,  we  are 
slowly  realizing  that  the  limited  repertoire  of  dynamical  behavior 
(fixed  points)  implemented  by  the  DCAMs  constrain  their  use  as 
information  processing  devices  for  signals  that  carry  information 
in  their  time  structure.  For  instance,  the  point  attractor  has  no 
dynamical  memory  (i.e.  the  system  forgets  all  previous  inputs 
when  it  reaches  the  fixed  point);  while  the  dynamic  memory  of 
the  limit  cycle  is  constrained  to  the  period;  only  chaotic  systems 
display  long  term  dynamic  memory  due  to  the  sensitivity  to  initial 
conditions.  This  sensitivity  carries  the  problem  of  susceptibility  to 
noise,  but  a  possible  solution  is  to  utilize  a  chaotic  attractor  cre¬ 


2.  MESOSCOPIC  DYNAMICS:  A  QUICK 
OVERVIEW 

We  will  be  modeling  brain  dynamics  at  the  level  of  million  of 
cells,  which  is  called  the  mesoscopic  level  [2].  The  advantage  of 
the  mesoscopic  level  is  that  it  is  able  to  describe  interactions 
among  sufficient  large  populations  of  cells  that  are  relevant  to 
probe  information  processing.  Action  potentials  limit  the  model¬ 
ing  to  relatively  small  number  of  cells,  while  the  electroencepha¬ 
logram  describes  too  global  interactions. 

Our  unit  of  processing  will  be  called  a  processing  element  (PE). 
PEs  are  modular  and  formal  representations  of  populations  of 
neurons  in  the  mammalian  brain.  The  representation  of  responses 
is  done  by  means  of  density  functions  that  measure  the  pulse  to 
wave  conversion,  e.g.  from  a  firing  pulse  density  of  a  set  of  neu¬ 
rons  to  the  current  density  and  back  to  a  pulse  density.  Hence,  the 
functions  are  continuous  in  time,  magnitude  and  space.  From  bio¬ 
logical  evidence  Freeman  as  proposed  the  following  KO  structure 
[1]  shown  in  figure  1  to  represent  a  population  of  neurons. 


F(s) 


Figure  1.-  KO  model  diagram  (excitatory). 

The  KO  PE  comprises  a  summing  input  node  where  the  spacial 
inputs  converge  into  a  order  dynamics  block  F(s)  represented 
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by  (1).  The  output  signal  is  the  result  of  a  pulse  to  wave  density 
conversion  function.  This  signal  is  then  reconverted  into  a  pulse 
density  signal  by  means  of  a  non-symmetric  non-linear  function 
represented  also  in  figure  1 .  This  is  the  wave  to  pulse  conversion 
phase  and,  in  terms  of  densities,  is  functionally  represented  hy  (2). 
Figure  2  shows  the  non-linearity  for  different  ‘q’  values. 


work  is  a  lattice  of  KII  PEs  that  models  the  olfactory  cortex.  The 
excitatory  connections  among  KII  PEs  are  modulated  by  Hebbian 
learning,  while  the  negative  connections  among  KII  PEs  are  fixed. 
Eunctionally  speaking,  the  KII  network  is  an  associative  memory, 
creating  a  mapping  from  input  patterns  to  a  distinctive  spatio  tem¬ 
poral  pattern  of  activity  for  each  input  over  the  coupled  lattice. 
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The  olfactory  cortex  is  a  hierarchical  arrangement  of  KII  net¬ 
works,  tightly  coupled  by  dispersive  delays.  In  our  Kill,  we 
model  the  olfactory  hulh  (OB),  the  anterior  olfactory  nucleus 
(AON),  the  prepiriform  cortex  (PC),  and  the  entorhinal  cortex 
(EC).  Since  the  individual  KII  oscillators  in  each  of  these  cortical 
areas  have  different  intrinsic  oscillating  frequencies  and  are 
tightly  coupled  by  delays  connections,  the  Kill  network  displays 
nonconvergent  (chaotic)  dynamics.  Hence,  it  is  locally  stable  but 
globally  unstable.  With  no  input,  the  Kill  model  lies  in  a  high 
dimensional  chaotic  state.  Once  it  is  activated,  the  system  dynam¬ 
ics  quickly  collapse  to  a  lower  dimensional  attractor  (an  “attractor 
wing”)  where  the  input  is  recognized.  When  the  input  is  taken  out, 
the  system  state  returns  to  the  high  dimensional  chaotic  regime, 
waiting  for  a  new  input.  The  information  processing  characteris¬ 
tics  of  this  design  have  not  been  thoroughly  investigated  yet. 


3.  ANALOG  VLSI  CIRCUIT  IMPLEMEN¬ 
TATION 


Figure  2.-  Non-symmetric  non-linearity  function  (eq. 
(2)). 


External 


Figure  3.-  KII  PE  representation  (each  square  represents 
a  KO  of  figure  1). 

The  KO  is  the  building  block  of  an  hierarchical  family  that  models 
different  types  of  anatomical  connections.  Eor  instance,  in  the 
olfactory  system  of  the  rabbit,  the  KO  network  models  the  periglo- 
mular  layer  (only  excitatory  cells).  The  KII  is  a  special  arrange¬ 
ment  of  KOs  forming  a  multi-input  multi-output  (MIMO) 
network.  The  arrangement  mimics  biological  evidence  and  is 
shown  in  figure  3.  The  (-I-)  and  (-)  signs  describe  excitatory  and 
inhibitory  connections.  The  inhibitory  KO  is  simply  obtained  by 
negating  the  output  of  figure  1  (flipping  the  sigmoidal  non-linear¬ 
ity  over  the  x  axis).  Under  correct  parameter  sets,  defined  as  inter¬ 
connection  gains,  the  KII  PE  behaves  as  an  oscillator  controlled 
by  the  input.  It  is  zero  for  zero  input  and  oscillates  if  the  input  (an 
external  signal  shown  in  figure  3)  amplitude  rises. The  KII  net¬ 


The  formulation  of  the  cell  assembly’s  behavior  in  terms  of  non¬ 
linear  dynamics  opens  the  door  to  analog  VLSI  simulations.  This 
is  one  of  the  advantages  of  the  nonlinear  dynamics  framework  for 
neuroengineering.  It  enables  direct  translation  of  brain  computa¬ 
tional  principles  into  engineering  systems,  even  when  we  do  not 
truly  understand  the  purposes  of  the  processing.  Conversely,  it 
enables  us  to  simulate  brain  processes  in  both  analog  and  digital 
computers,  helping  us  derive  and  check  mathematical  hypothesis 
that  furthers  our  understand  of  the  brain. 

We  will  not  present  in  this  paper  the  design  of  the  analog  VLSI 
components  that  were  developed  to  implement  the  KO  PE.  Please 
consult  the  following  papers  [5], [6].  We  will  only  briefly  address 
the  design  principles  that  were  utilized  and  present  results  of  our 
chips.  Designing  VLSI  chips  to  mimic  brain  function  is  a  formi¬ 
dable  problem  basically  for  three  reasons:  power  consumption, 
size  of  the  components  and  the  massive  interconnect.  We  utilized 
analog  VLSI  in  suhthreshold  regimes  to  decrease  the  power  con¬ 
sumption  and  the  size.  We  utilized  hybrid  (analog  amplitude/ 
discrete  time)  components  to  fully  utilize  the  power  of  analog 
computation  (time  is  a  free  parameter  for  the  computational  algo¬ 
rithms),  while  enabling  us  to  multiplex  signals  and  save  real  state 
for  the  interconnect. 

In  order  to  implement  a  KO  PE  one  needs  to  design  an  adder,  a 
second  order  dynamic  element  (a  lowpass  filter  with  specified 
time  constants),  and  a  nonsymmetric  static  nonlinearity.  Once  this 
is  accomplished,  the  KII  is  an  arrangement  of  KO  and  the  Kill  is 
an  arrangement  of  KII  connected  by  dispersive  delays.  The  KII 
setup  for  our  tests  is  shown  in  figure  4,  with  table  I  summarizing 
the  parameter  values.  The  results  from  an  actual  VLSI  implemen¬ 
tation  are  plotted  in  figures  5  and  6.  As  expected  the  KII  behaves 
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as  an  oscillator  controlled  by  the  input.  The  waveform  phases 


Figure  6.-Measured  KII  phase  plot  showing  a  complete 
cycle  correspondent  to  figure  11.  All  KO  cells  in  figure  10 


Figure  4.-  KII  schematic  (Rp  and  Ry;  j,i-E  or  I,  corre-  oscillate  at  the  same  frequency  but  out  of  phase. 


spond  to  Rsp;  k=l,..,n  in  figure  7,  respectively). 


Table  1:  KII  Interconnections  gain 


Gain 

Value 

Rf^ee 

2.5 

Rf/Rii 

2.5 

Rf^ie 

2 

Rf^ei 

5 

As  can  be  seen  in  figure  6,  the  state  of  a  KII  PE  has  two  sta¬ 
ble  regimes:  a  fixed  point  at  the  origin,  and  a  limit  cycle. 
The  transition  between  these  two  points  is  controlled  by  the 
input.  This  attractor  has  been  called  a  Shilkonov  attractor. 
When  a  20  bit  input  stream  is  presented  to  a  KII  network, 
and  the  excitatory  connections  are  modified  with  Hebbian 
learning  the  OB  PEs  display  oscillations  of  different  ampli¬ 
tudes,  coding  “1”  as  high  amplitude,  and  “0”  as  low  ampli¬ 
tude  oscillations  as  shown  in  Figure  7. 


1("0")  2r0"l  3C0")  4ro-)  5("0") 


6("0"1  7r0")  81-0")  9rr)  10(-l") 


0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9  1 

Time  (s) 

Figure  5.-  Measured  KII  oscillatory  behavior  with  the 
input. 


11  ("0”)  12("0")  13{’’0")  14(''1")  15("0”) 


16("0”)  17("00  18("1")  19(”0'’)  20("1”) 


Figure  7.  Oscillatory  patterns  over  a  20  PE  OB  layer. 
Channels  that  have  a  high  amplitude  codify  “1”  in  the 
input  pattern. 

We  still  do  not  have  chip  simulations  for  the  Kill  network. 
Our  simulation  of  the  dynamics  using  digital  signal  pro¬ 
cessing  techniques  [4],  show  that  in  fact  the  Kill  model  has 
nonconvergent  dynamics,  possibly  chaotic  [4].  Instead  of 
sinusoids  of  different  amplitudes,  signals  have  a  much 
more  complex  time  structure,  but  are  still  modulated  in  the 
same  way  (i.e.  KII  PEs  that  codify  “I”  have  higher  ampli¬ 
tude).  Figure  8  shows  phase  plots  of  a  Kill  network  simula¬ 
tion  taken  at  different  points  in  the  hierarchy.  We  see 
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trajectories  that  are  compatible  with  a  chaotic  attractor,  as 
discussed  by  Freeman  [7]. 


OB  Channel  2 


Figure  8.  Phase  plots  at  different  levels  of  the  Kill 
hierarchical  model  of  the  olfactory  cortex  without  an 
input,  simulated  in  a  digital  computer.  The  plots  are 
compatible  with  a  chaotic  attractor. 


4.  CONCLUSION 
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Letters,  Vol.  35  No  15,  July  1999 
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This  paper  starts  by  posing  the  important  question  of  how  to  trans¬ 
late  signals  into  symbols  preserving  maximally  the  information  in 
the  signal  time  structure.  From  biological  studies,  it  seems  that  the 
brain  of  mammals  solves  the  problem  by  coupling  the  sensory 
inputs  to  locally  stable  but  globally  unstable  dynamical  lattices  of 
rather  simple  nonlinear  dynamical  processing  elements.The  lan¬ 
guage  of  dynamics  helps  us  simulate  and  implement  in  analog 
VLSI  circuits  with  similar  dynamics.  Hence,  this  opens  up  the 
possibility  of  creating  brain  dynamics  in  chips  that  can  be  used  to 
create  animats  which  may  display  the  same  sensitivity,  selectivity, 
adaptiveness,  speed,  and  tolerance  to  noise  that  characterizes 
human  sensation. 
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